Skip to content

feat(cloudfetch): honor download settings with the kernel - #566

Open
aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114
Open

aakash-saravanan-db wants to merge 6 commits into
mainfrom
aakash-saravanan_data/feat/cloudfetch-download-settings-20261114

Conversation

@aakash-saravanan-db

@aakash-saravanan-db aakash-saravanan-db commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Feature or gap

Go applications could set WithMaxDownloadThreads or maxDownloadThreads in the DSN, but kernel connections ignored the setting.

What this PR addresses

Forward explicit standard settings into the existing kernel download window; keep the kernel default of 16 when unset. Non-positive explicit values normalize to 10. Preserve explicit kernel/raw overrides and document the 256 ceiling, connection timing and additional consumer buffers. A positive WithKernelMaxChunksInMemory writes the canonical lowercase key; the kernel applies case variants in sorted order, so it takes precedence over raw spellings and strips all of them before server requests.

Dependencies

Existing native kernel APIs; no native API or version bump is required for this mapping.

Tests and validation

CGO_ENABLED=0 go test ./... passes. Built-driver checks verify explicit settings, DSN parsing, precedence, the unset default, complete results and reuse. After equal warmups and five alternating runs per side, using a native built from kernel a488a05, DBSQL submission-through-drain medians for 3M ordered rows: Thrift 6.33 s → kernel 4.15 s (-34.4%).

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good: one low test-coverage gap. The kernel's forwarding precedence (positive explicit option > case-insensitive raw session parameter > MaxDownloadThreads when set) is implemented correctly in buildKernelConfig, DeepCopy carries the new flag, and values are normalized before forwarding. Possible follow-up, not caused by this PR: when both an uppercase raw key and WithKernelMaxChunksInMemory are set, both keys still reach the kernel.

query string
set bool
}{
{"", false},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Low — TestMaxDownloadThreadsDSNPresence only has cases where set is false ("" and ?maxDownloadThreads=), so the if tc.set { assert.Equal(t, "10", value) } branch never runs. That leaves the path this PR adds untested: DSN maxDownloadThreads=N → MaxDownloadThreadsSet → cloudfetch_max_chunks_in_memory=N in the kernel SessionConf. The DSN subtest in TestMaxDownloadThreadsNonPositiveDegradesToDefault asserts the flag but never calls buildKernelConfig. Add a set case such as {"?maxDownloadThreads=4", true} and assert the forwarded value (the hard-coded "10" would then also need to follow the case's value), or a ?maxDownloadThreads=0 case that expects "10".

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good. There is one low finding: the CHANGELOG entry doesn't flag that existing maxDownloadThreads settings now change kernel memory and throughput. The forwarding logic matches what the docs describe and the tests cover it: an explicit value is forwarded, the default of 16 stays when unset, a raw session parameter in any letter case takes precedence, WithKernelMaxChunksInMemory takes precedence over both, and the caller's map is not changed.

Comment thread CHANGELOG.md Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Looks mostly good, with 1 medium concern: when a positive WithKernelMaxChunksInMemory is combined with a raw chunk session parameter cased differently (e.g. uppercase), both keys reach the kernel, so the documented precedence isn't enforced. Forwarding maxDownloadThreads via the new MaxDownloadThreadsSet flag, the DSN/option presence tracking and DeepCopy handling all look correct and are covered by tests.

Comment thread kernel_config.go Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Mostly good, with one low concern: the documented case-insensitive precedence isn't enforced when an explicit override and a mixed-case raw key are both set. Forwarding maxDownloadThreads only when it is explicitly set, plus the DSN presence tracking, looks right and is well tested. Nit: the telemetry MaxChunksInMemory comment was reworded, but the field is still never populated.

Comment thread kernel_config.go Resolved

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Nit

Looks good: no correctness concerns found. I traced how the setting flows through the code. MaxDownloadThreadsSet is set by both the option and ParseDSN; an empty DSN value leaves it unset because extractAsInt only reports values that are present and non-empty. withUserConfig and DeepCopy carry it through. Non-positive values are changed to 10 in NewConnector before buildKernelConfig runs. The precedence is checked against session-parameter keys that differ only in case, and the explicit kernel option is still applied last. One trade-off is already documented in the CHANGELOG: an explicit maxDownloadThreads=10 (common in sample DSNs) now lowers the kernel window from 16 to 10. The same applies to Reyden auto-fallback connections.

Other findings

  • ⚪ Nit — telemetry/request.go: MaxChunksInMemory is never filled in by the driver (the kernel owns telemetry), so the new comment "zero means the value was not reported" is accurate. It could also say the effective window may now come from maxDownloadThreads.
  • ⚪ Nit — kernel_config_test.go: the rewritten "unset download window" subtest only covers KernelExperimental set to a non-positive value. The old case with nil KernelExperimental is now covered only indirectly, by TestMaxDownloadThreadsDSNPresence. Adding the nil case back to the loop would keep the guard local.

Comment thread connector.go
func WithMaxDownloadThreads(numThreads int) ConnOption {
return func(c *config.Config) {
c.MaxDownloadThreads = numThreads
c.MaxDownloadThreadsSet = true

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this in Go but not Python and Node?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go initializes MaxDownloadThreads to 10 even when omitted, so this flag distinguishes “unset” from “explicitly set to 10.”

Python can detect omission through its keyword arguments. Node tracks explicit assignments through its config property accessor.

All three now preserve the kernel default of 16 when unset and honor explicit values, including 10. The flag is Go’s way of preserving that distinction.

Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
@aakash-saravanan-db
aakash-saravanan-db force-pushed the aakash-saravanan_data/feat/cloudfetch-download-settings-20261114 branch from ac2504c to 03c875f Compare November 17, 2026 06:03

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@aakash-saravanan-db aakash-saravanan-db added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants